Lecture 6

class: center, middle, inverse, title-slide

# Lecture 6
## Model Evaluation
### Psych 10 C
### University of California, Irvine
### 04/11/2022

---

## Summary

- Last week we started with two different hypotheses about the relation between
lung capacity and smoking status.

- The Null model stated that there where no differences in lung capacity as a 
function of smoking status, the model was formalized as:
`$$y_{ij}\sim\text{Normal}(\mu,\sigma^2)$$`

- We found two estimators for the parameters in the model, `$\hat{\mu}$` which is 
the average of the participants lung capacity regardless of smoking status.

- And `$\hat{\sigma}^2_0$` which is the error or variability of our observations 
when we use `$\hat{\mu}$` as a prediction of each observation.

- Finally we said that we would be interested on the Sum of Squared Errors of the
Null model which is defined as: 
`$$SSE_0 = \sum_j \sum_i \left(y_{ij}-\hat{\mu}\right)^2$$`

---

## Summary

- Our second hypothesis was the Effects Model, which assumes that there is a 
difference in lung capacity as a function of smoking status. This model is 
formalized as:

`$$y_{ij}\sim\text{Normal}(\mu_j,\sigma_e^2)$$`

- The estimator of our parameter `$\mu_j$` (one for each group `$\hat{\mu}_j$`) was 
equal to the average of each group (taken independently).

- The estimator for `$\sigma_e^2$` was equal to the error of the model, which is 
the average squared difference between each observation and the model's 
prediction for that observation `$\hat{\mu}_j$`.

- Finally, we mention that we will also be interested in the Sum of Squared 
Errors of the Effects Model, which is defined as:
`$$SSE_e = \sum_j \sum_i \left(y_{ij}-\hat{\mu}_j\right)^2$$`

---

## Adding predictions

- The firs thing that we want to do is add our predictions and the 
squared error of each observation to our data.

```r
# total sample size
n_total <- nrow(smokers)
# get the predictions of effects model (\hat{\mu}_j)
eff_pred <- smokers %>% 
  group_by(smoke_status) %>% 
  summarise("prediction" = mean(lung_capacity))
# add predictions to data
smokers <- smokers %>% 
  mutate("pred_null" = rep(x = mean(lung_capacity), times = n_total),
         "pred_eff" = ifelse(test = smoke_status == "smoker",
                             yes = eff_pred$prediction[2], 
                             no = eff_pred$prediction[1]))
```

---

## Adding errors

- Now to add the squared errors we can use the difference between prediction and
observation squared:

```r
smokers <- smokers %>% 
  mutate("error_null" = (lung_capacity - pred_null)^2,
         "error_eff" = (lung_capacity - pred_eff)^2)
```

- Now our data file has the relevant variables for each observation:

<div id="htmlwidget-9823eb348fc443cf78ff" style="width:100%;height:auto;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-9823eb348fc443cf78ff">{"x":{"filter":"none","vertical":false,"fillContainer":false,"data":[["1","2","3","4","5","6","7","8"],[78.3,80,73.6,77.4,68,64.4,70.8,65.5],[72.25,72.25,72.25,72.25,72.25,72.25,72.25,72.25],[77.33,77.33,77.33,77.33,67.17,67.17,67.17,67.17],[36.6,60.06,1.82,26.52,18.06,61.62,2.1,45.56],[0.95,7.16,13.88,0.01,0.68,7.7,13.14,2.81]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>lung_capacity<\/th>\n      <th>pred_null<\/th>\n      <th>pred_eff<\/th>\n      <th>error_null<\/th>\n      <th>error_eff<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":4,"columnDefs":[{"className":"dt-right","targets":[1,2,3,4,5]},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[4,10,25,50,100]}},"evals":[],"jsHooks":[]}</script>

---

## Sum of Squared Errors

- Using the updated table it's easy to get the values of the SSE and the 
estimators `$\hat{\sigma}_0^2$` and `$\hat{\sigma}_e^2$`.

```r
# the sse of the null model is:
sse_0 <- sum(smokers$error_null)
# the sse for the effects model is:
sse_e <- sum(smokers$error_eff)
# mean sse null model
sigma_0 <- 1/n_total * sse_0
# mean sse effects model
sigma_e <- 1/n_total * sse_e
```

Their values are:

.pull-left[
Null Model

`$SSE_0$` = 252.36

`$\hat{\sigma}_0^2$` = 31.54
]

.pull-right[
Effects Model

`$SSE_e$` = 46.31

`$\hat{\sigma}_e^2$` = 5.79
]

---

class: inverse, center, middle

# Model Evaluation

## `$R^2$`

---

## Model Evaluation: `$R^2$`

- `$R^2$` is a method that allows us to "measure" how good a general model is in 
comparison to a **nested** model.

- In our problem, the Null model is nested on the Effects model, but what does 
it mean for a model to be nested?

- We say that one model is nested on another when the values that the parameters
of the nested model are a special case of the second.

- For example, in our effects model there is nothing that makes it impossible 
for `$\mu_1$` to be equal to `$\mu_2$`, which would be the same as saying that the
expectation of each group was the same...

- But that was the assumption of the **Null Model**!

- This means that the Null model is nested in the Effects Model, in other words, 
the **Null Model** is a special case of the **Effects Model**.

---

## Model Evaluation: `$R^2$`

- What does it mean for our results for the Null Model to be nested on the 
Effects Model?

- Well this means that, in terms of error or variability, the Effects Model 
will always have a lower than or equal error than the Null Model.

- In other words, the worse that the Effects Model can do is to be equal to the 
Null Model, everything else will make the Effects Model better.

- We can express this new information formally using the following equation:
`$$SSE_0 = SSE_a + SSE_e$$`

- Where the new variable `$SSE_a$` represents the error or variation that is 
reduced or accounted for when we use the Effects Model In comparison to the 
Null.

- Given that we know that `$SSE_e$` will always be **equal or slower** than `$SSE_0$`
we can express the error accounted for by the Effects Model as:
`$$SSE_a = SSE_0 - SSE_e$$`

---

## Model Evaluation: `$R^2$`

- `$R^2$` can be interpreted as the proportion of error accounted for by the 
effects model out of the total error.

- The total error is the error of the Null model, and the error accounted for is 
`$SSE_a$`. Therefore, we can express the proportion of accounted error `$(R^2)$` as:
`$$R^2 = \frac{SSE_a}{SSE_0} = \frac{SSE_0 - SSE_e}{SSE_0}$$`

- By definition we have that: `$$0 \leq R^2 \leq 1$$`

---

## Model Evaluation: `$R^2$`

- When `$R^2 = 0$` there is no difference in the error (variance) of the models.
  
--

- When `$R^2 = 1$` it means that the error (variance) of the Effects model is 0 (which will can't happen unless that our observations have no variability).

- This means that the closer the value of `$R^2$` is to 1 the better the effects 
model is!

- Whenever you see a Journal paper that uses a linear model it is likely that 
you will encounter a value of `$R^2$` defined as the proportion of error accounted
for by the Effects model (or linear model).

- Let's calculate the proportion of error accounted for in our smokers example.

---

## `$R^2$`: Smokers Experiment

- Using the Sum of Squared Errors that we calculated before we can obtain the 
proportion of error accounted for by the effects model.

- Go back to slide 6 and using the values there calculate the proportion of 
error accounted for by the effects model in the smokers data set.

<br>

.can-edit.key-likes[
`$R^2 =$`
]

---

## `$R^2$`

- `$R^2$` will give us a first look at how the Effects model is doing in a given 
situation.

- However, there is a problem with `$R^2$` that can't be avoided...

- Remember that we said that the error (variance) of the Effects model will 
always be equal or lower than that of the Null model.

- This is because using two distributions instead of one gives us more 
flexibility.

- In other words, we can always do better using two normal distributions than usin 
a single one as in the Null model.